Runtime Optimization of Continuous Queries

In data stream processing systems, Quality of Service (or QoS) requirements, as specified by users, are extremely important. Unlike in a database management system (DBMS), a query in a data stream management system (DSMS) cannot be optimized once and executed. It has been shown that different scheduling strategies are useful in trading tuple latency requirements with memory and throughput requirements. In addition, DSMSs may experience significant fluctuations in input rates. In order to meet the QoS requirements of data stream processing, a runtime optimizer equipped with several scheduling and load shedding strategies is critical. This entails monitoring of QoS measures at run-time to dynamically modify the processing of the queries at runtime to meet the QoS requirements. This paper addresses runtime optimization issues for MavStream, a data stream management system (DSMS). The runtime optimizer presented in this paper matches the output (latency, memory, and throughput) of a continuous query (CQ) with its QoS requirements. Alternative scheduling strategies are chosen as needed based on the runtime feedback. A decision table is used to choose a scheduling strategy based on the priorities of QoS requirements and their violation. The decision table approach allows us to add new scheduling strategies as well as compute the strategy to be used in an extensible manner. Additionally, load shedders are activated and deactivated by the runtime optimizer to meet QoS requirements beyond adjusting scheduling strategies..